{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why we use `Naive bayes` ?\n",
    "\n",
    "**Step 1** - We are trying to calculate what is probability of class 1 (other will be just 1- p(1)) given a document x. i.e. P(c=1|x).  \n",
    "**Step 2** - Using naive bayes formula, we can rewrite it as: P(c=1|x) = {P(x|c=1) P(c=1)} / P(x) . Why we did this transformation? Because from data we know that how many of total docs are class = 1, and how many of class 1 have words that we have in our document. We can rewrite -> P(x|c=1) as P(x1|c=1) P(x2|c=1) P(x3|c=1) if x1,x2,x3 are 3 words in our doc x (considering independence of x1,x2,x3). But we can not rewrite, P(c=1|x) as P(c=1|x1) P(c=1|x2) P(c=1|x3). So just a mathematical transformaiton to easily calculate probabilities.   \n",
    "\n",
    "**Step 3** So from data (term doc matrix), what we need are no. of times class = 1, no. of times some word appear whereever class = 1. (same with class 0 too)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'fastai.models'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-c6b923bd0731>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mfastai\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnlp\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinear_model\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mLogisticRegression\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mtorchtext\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mvocab\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdatasets\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/fastai/lib/python3.6/site-packages/fastai/nlp.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mimports\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mtorch_imports\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mcore\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/fastai/lib/python3.6/site-packages/fastai/torch_imports.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mtorchvision\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mdensenet121\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdensenet161\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdensenet169\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdensenet201\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresnext_50_32x4d\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mresnext_50_32x4d\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresnext_101_32x4d\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mresnext_101_32x4d\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresnext_101_64x4d\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mresnext_101_64x4d\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'fastai.models'"
     ]
    }
   ],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import ParameterGrid\n",
    "from joblib import Parallel, delayed\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
    "\n",
    "\n",
    "from fastai.nlp import *\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from torchtext import vocab, data, datasets\n",
    "from IPython.core.display import Image "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#! ls /Users/groverprince/Documents/msan/msan_ml/data/aclImdb/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sentiment classification task consists of predicting the polarity (positive or negative) of a given text.\n",
    "To get the dataset, in your terminal run the following commands:  \n",
    "`wget http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz`   \n",
    "`gunzip aclImdb_v1.tar.gz`  \n",
    "`tar -xvf aclImdb_v1.tar`  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH='/Users/groverprince/Documents/msan/msan_ml/data/aclImdb/'\n",
    "names = ['neg','pos']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "desk = '/Users/groverprince/Desktop/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn, trn_Y = texts_from_folders(f'{PATH}train', names) # taking text from each file, saving in list. saving 0 or 1 in other list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "val,val_y = texts_from_folders(f'{PATH}test',names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"Story of a man who has unnatural feelings for a pig. Starts out with a opening scene that is a terrific example of absurd comedy. A formal orchestra audience is turned into an insane, violent mob by the crazy chantings of it's singers. Unfortunately it stays absurd the WHOLE time with no general narrative eventually making it just too off putting. Even those from the era should be turned off. The cryptic dialogue would make Shakespeare seem easy to a third grader. On a technical level it's better than you might think with some good cinematography by future great Vilmos Zsigmond. Future stars Sally Kirkland and Frederic Forrest can be seen briefly.\",\n",
       " 0,\n",
       " 25000)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn[0], trn_Y[0], len(trn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"Once again Mr. Costner has dragged out a movie for far longer than necessary. Aside from the terrific sea rescue sequences, of which there are very few I just did not care about any of the characters. Most of us have ghosts in the closet, and Costner's character are realized early on, and then forgotten until much later, by which time I did not care. The character we should really care about is a very cocky, overconfident Ashton Kutcher. The problem is he comes off as kid who thinks he's better than anyone else around him and shows no signs of a cluttered closet. His only obstacle appears to be winning over Costner. Finally when we are well past the half way point of this stinker, Costner tells us all about Kutcher's ghosts. We are told why Kutcher is driven to be the best with no prior inkling or foreshadowing. No magic here, it was all I could do to keep from turning it off an hour in.\",\n",
       " 0,\n",
       " 25000)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val[0],val_y[0], len(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this is just like \"this is good\" --> this = 1, is = 1, good = 1\n",
    "# but it is just an object\n",
    "\n",
    "veczr = CountVectorizer(tokenizer=tokenize) # making tokenizer (bag of words representation) - sparse matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using veczr object to make term doc matrix\n",
    "\n",
    "trn_term_doc = veczr.fit_transform(trn)  \n",
    "val_term_doc = veczr.transform(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "128"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(val[0].split()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<1x75132 sparse matrix of type '<class 'numpy.int64'>'\n",
       " \twith 93 stored elements in Compressed Sparse Row format>,\n",
       " <25000x75132 sparse matrix of type '<class 'numpy.int64'>'\n",
       " \twith 3640465 stored elements in Compressed Sparse Row format>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc[0], val_term_doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 25000 = number of training documents . \n",
    "* 75132 = number of unique words (represent columns of tokenized version) in all documents combined . keys of dict\n",
    "* training term document has 93 words out of those 75132 unique words. \n",
    "* 128 should technically have been 93. maybe because of ./', signs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1205.7655009765356"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "100000*(146**-0.75)*(np.log10(1+np.log10(162)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get `feature_names` in particular column index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ameteurish', 'ametuer', 'amfortas', 'amg', 'ami']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = veczr.get_feature_names(); vocab[3000:3005]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "opposite of above, to get index of particular `column name (vocublary)`. _ is just used to replace inplace (not here) what is it for ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3001"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "veczr.vocabulary_['ametuer']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "what is going on here?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc[0,2297]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use of Naive Bayes to fit a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=trn_term_doc\n",
    "y=trn_Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 75132)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape # 75312 columns, distinct words, 25000 documents/rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = x[y==1].sum(0)+1  # no of times a word appear, given class = 1. \n",
    "q = x[y==0].sum(0)+1  # no of times a word appear, given class = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1, 75132), (1, 75132))"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p.shape, q.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(matrix([[    2,     1, 11820, ...,     2,     2,     1]], dtype=int64),\n",
       " matrix([[    1,     2, 12742, ...,     1,     2,     8]], dtype=int64))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p, q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "p/psum =  (total times word a1 appear in all docs)/(total words appearning in all the docs) (+ 1 is added for each word, ignoring that for understanding for now). (all this given class = 1)\n",
    "What does it mean? It means, given class = 1, what is probability of word a1 appearing. \n",
    "\n",
    "we calculate ratio of this wrt when class = 0 as ratio gives whether this word has higher chance of being from class 1 or 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(matrix([[3747495]], dtype=int64), matrix([[3789022]], dtype=int64))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q.sum(1) , p.sum(1) # total words in 0 class are more"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1, 75132),\n",
       " matrix([[ 0.68213, -0.70417, -0.08613, ...,  0.68213, -0.01102, -2.09046]]))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = np.log((p/p.sum(1))/(q/q.sum(1))) \n",
    "r.shape, r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1, 75132),\n",
       " matrix([[ 0.69315, -0.69315, -0.07511, ...,  0.69315,  0.     , -2.07944]]))"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = np.log(p/q)\n",
    "b.shape, b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "by using naive bayes prediction, we are sort of directly making a model and those predictions can help us to predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(75132, 1)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.T.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<25000x75132 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 3640465 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_term_doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "validation has 1 or no. of times occurence of word in doc where that word appears, 0 otherwise. so it will multiply `word count in doc with chances of word appearning given class 1 probability is higher` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "pre_preds = val_term_doc @ r.T "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ -8.33253, -22.4765 , -26.97431, ...,  97.91732,  18.31415,  21.76217]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pre_preds.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.80740000000000001"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pre_preds = val_term_doc @ r.T\n",
    "preds = pre_preds.T>0  # because if sum of log ratios is > 0 means class 1\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "binarised naive bayes. Funda is don't need to take number of occureces of word in a document. Just whether is appears or not is fine. Think of a case where a word appears many many times but only in 1 document for class 1 and once in many documents of class 0. Where should it be assigned. Class 1 or 0? Here it is trying to assign it to class 0 which makes more sense and that appearance in only 1 doc for class 1 might just be an outlier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.82623999999999997"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pre_preds = val_term_doc.sign() @ r.T + b\n",
    "preds = pre_preds.T>0\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ -4.35523, -19.08058, -23.69877, ...,  54.90807,  14.71441,  18.40103]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pre_preds.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ 0.68213, -0.70417, -0.08613, ...,  0.68213, -0.01102, -2.09046]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try same thing using **Logistic regression**. (Considering onyl unigrams like this first)\n",
    "\n",
    "`C` in Logistic is inverse of regularization constant ($ \\lambda $). So lower c means high regularization. Let's try first without any regularization. Then we will move on to regularization to see how it improves (by regularizing parameters we aim to hangle overfitting)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.85387999999999997"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C = 1e5)\n",
    "m.fit(x,y)  # feeding raw form of sparse matrix and target y without changin anything -- so easy :D\n",
    "preds = m.predict(val_term_doc)\n",
    "np.mean(preds == val_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing that we tried in NBC, we can try here too. Ignore no. of times a word appears in document. Even if it word `a1` appears 5 times in a doc, just take 1. That what `val_term_doc.sign()` can do. It return 1 for +ve, 0 for 0 and -1 for all negative values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.85768"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C = 1e5)\n",
    "m.fit(x.sign(),y)  # sign() in training data\n",
    "preds = m.predict(val_term_doc.sign())  # sign in test data\n",
    "np.mean(preds == val_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, that's observable! binarising our matrix values improves accuracy. .85768 > .85387"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try some regularization. (maybe we are overfitting). But let's first check if we are actually overfitting or not. Way to do that is to check both training and validation score. If training score is very high, then there is chance of overfitting. Let's check that"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = m.predict(x.sign())\n",
    "np.mean(preds == y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow. training accuracy is 100%. Definitely it is overfitting. Let's reduce it by introducing some regularization. \n",
    "Starting with `c = 0.1` first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.88400000000000001, 0.96736)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C = 0.1)\n",
    "m.fit(x.sign(),y)  # sign() in training data\n",
    "preds = m.predict(val_term_doc.sign())  # sign in test data\n",
    "preds2 = m.predict(x.sign())\n",
    "\n",
    "np.mean(preds == y), np.mean(preds2 == y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. So our validation accuracy has definitely improved using some regularization and training has reduced too. Let's try to tune `C` to find best value of it by using a gridsearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fitOne(model, X, y, params):\n",
    "    m = model(**params)\n",
    "    return m.fit(X, y)\n",
    "\n",
    "def fitModels(model, paramGrid, X, y, n_jobs=-1, verbose=10):\n",
    "    return Parallel(n_jobs=n_jobs, verbose=verbose)(delayed(fitOne)(model,\n",
    "                                                                    X,\n",
    "                                                                    y,\n",
    "                                                                    params) for params in paramGrid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done   1 tasks      | elapsed:    1.1s\n",
      "[Parallel(n_jobs=-1)]: Done   2 out of   7 | elapsed:    1.9s remaining:    4.8s\n",
      "[Parallel(n_jobs=-1)]: Done   3 out of   7 | elapsed:    2.8s remaining:    3.8s\n",
      "[Parallel(n_jobs=-1)]: Done   4 out of   7 | elapsed:    4.6s remaining:    3.5s\n",
      "[Parallel(n_jobs=-1)]: Done   5 out of   7 | elapsed:    7.7s remaining:    3.1s\n",
      "[Parallel(n_jobs=-1)]: Done   7 out of   7 | elapsed:   10.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Done   7 out of   7 | elapsed:   10.0s finished\n"
     ]
    }
   ],
   "source": [
    "model = LogisticRegression\n",
    "grid = {\n",
    "    'C': [1e-4, 1e-3, 1e-2, 1e-1, 1, 10, 100], # regularization\n",
    "    'penalty': ['l2'], # penalty type\n",
    "    'n_jobs': [-1] # parallelize within each fit over all cores\n",
    "}\n",
    "paramGrid = ParameterGrid(grid)\n",
    "myModels = fitModels(model, paramGrid, x.sign(), y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_score(m):\n",
    "    preds = m.predict(val_term_doc.sign())  # sign in test data\n",
    "    preds2 = m.predict(x.sign())\n",
    "\n",
    "    print(np.mean(preds == y), np.mean(preds2 == y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.80576 0.81452\n",
      "0.85056 0.8644\n",
      "0.8798 0.91468\n",
      "0.884 0.96736\n",
      "0.87384 0.99784\n",
      "0.86824 1.0\n",
      "0.8638 1.0\n"
     ]
    }
   ],
   "source": [
    "for i in myModels:\n",
    "    print_score(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In above print, lhs is validation score and rhs is train score. Validation score is highest for `C` = 0.1 and we can chose 0.1 as our regularization "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What else can be tried to improve the accuracy? One thing that can be tried is tweaking Naive bayes a little bit. We assumed our words/features to be independent of each other. Let's try some pair interaction that is something like making new features which are made using combinations of old independent features.  \n",
    "\n",
    "We can use `bigrams` or `trigrams`. What are `bigrams`. Very simple concept. Some words tend to appear together very frequently. Like `this is`, `what are`, `you are` etc. One major drawback of `naive` bayes (that's why it is called naive :P) is that it assumes all our features to be independent. But in actual sense they are not. Like frequenct occuring `what are`. these will come together and be dependent of each other. \n",
    "\n",
    "Bigram example -- `what are you doing` -> `what are`, `are you`, `you doing` . Adding bigrams will add both sinle word double adjacent word features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ngram_range=(1,2) does unigram, bigram and trigram. \n",
    "# max_features is limiting documents to not make each and every combination of adjacent 3 features. \n",
    "# It first sorts tri/bi grams based on number of together occurences and only takes ngrams which appear most.\n",
    "\n",
    "veczr =  CountVectorizer(ngram_range=(1,3), tokenizer=tokenize, max_features=800000)\n",
    "trn_term_doc = veczr.fit_transform(trn)\n",
    "val_term_doc = veczr.transform(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((25000, 800000), (25000, 800000))"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc.shape, val_term_doc.shape # 25k rows and max_feature (800k) columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's see how vocublary look like now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = veczr.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['! the camera',\n",
       " '! the cast',\n",
       " '! the character',\n",
       " '! the characters',\n",
       " '! the cinematography',\n",
       " '! the climax',\n",
       " '! the dialog',\n",
       " '! the direction']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[1002:1010]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "these words tend to appear together a lot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# repeating same steps to extract x, y \n",
    "\n",
    "x = trn_term_doc.sign()\n",
    "y = trn_Y\n",
    "val_x = val_term_doc.sign()\n",
    "\n",
    "p = x[y == 1].sum(0) + 1  # i need sum of words appearing in class 1. i.e in all docs\n",
    "q = x[y == 0].sum(0) + 1\n",
    "r = np.log((p/p.sum()) / (q/q.sum())) # r is log count ratio. log of count ratios\n",
    "b = np.log((y==1).sum()/(y==0).sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.88044"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# naive bayes again\n",
    "\n",
    "pre_preds = val_term_doc.sign() @ r.T\n",
    "preds = pre_preds.T>0\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.90500000000000003"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C=0.1, dual=True)\n",
    "m.fit(x, y);\n",
    "\n",
    "preds = m.predict(val_x)\n",
    "(preds.T==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the idea is to fit a logistic regression model with updated input matrix. We are updating input x to x.mult.r where r is log count ratio. Why? So, in NB solution x@r.T was our output (matrix mult) which was saying that `r` is a good weight matrix for our linear layer. Now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<25000x800000 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 12589101 stored elements in COOrdinate format>"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.multiply(r)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ -19.31273],\n",
       "        [-322.31574],\n",
       "        [ -36.51194],\n",
       "        ..., \n",
       "        [ 101.11562],\n",
       "        [  99.71359],\n",
       "        [  42.95914]])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x@r.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91768000000000005"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_nb = x.multiply(r)\n",
    "\n",
    "m = LogisticRegression(dual=True, C=0.1)\n",
    "m.fit(x_nb, y);\n",
    "\n",
    "val_x_nb = val_x.multiply(r)\n",
    "\n",
    "preds = m.predict(val_x_nb)\n",
    "(preds.T==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy improved from the previous version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### fasti NB SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "si = 2000 #(stop iteration)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "??TextClassifierData"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "??dotprod_nb_learner() # gets model . dot product with naive bayes like in above case "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is how we get a model from a bag of words\n",
    "md = TextClassifierData.from_bow(trn_term_doc, trn_Y, val_term_doc, val_y, si)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6487dca33f1547489d107b13992a7b34",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.02432  0.11949  0.91768]                         \n",
      "\n"
     ]
    }
   ],
   "source": [
    "learner = md.dotprod_nb_learner()  # \n",
    "learner.fit(0.02, 1, wds=1e-6, cycle_len=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ae4bd499be00469eb2a7afb1199d4215",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=2), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.01933  0.11368  0.92148]                         \n",
      "[ 1.       0.01358  0.1118   0.92104]                          \n",
      "\n"
     ]
    }
   ],
   "source": [
    "learner.fit(0.02, 2, wds=1e-6, cycle_len=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** 92.1% accurately predict whether IMDB review is good or bad**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
